Refactor Validation and Create RollCall module#56
Conversation
Move following functions: validate_csv_extension validate_output_directory validate_filename_uniqueness handle_missing_filenames setup_expected_columns
egrace479
left a comment
egrace479
left a comment
There was a problem hiding this comment.
Found a holdover from prior implementation method.
Co-authored-by: Elizabeth Campolongo <38985481+egrace479@users.noreply.github.com>
- Change tests - Include check_existing_images
This reverts commit ff930b0.
codedbyishika
changed the title
egrace479
left a comment
egrace479
left a comment
There was a problem hiding this comment.
I haven't had a chance to check the test changes, but some efficiency can be added and I have some questions in implementation of new features that rely on consideration of the motivation. Mainly the new duplicate check that should have an option to ignore for the purposes of still downloading, but then what is the intended response to duplicates; if the ignore flag is passed, should it be skipped or at least provide the duplicates in a usable format so the user can do something with the information?
| roll_call.check_duplicate_checksums( | ||
| data_df=source_df, | ||
| hash_col=args.verifier_col.lower(), | ||
| ignore_duplicates=getattr(args, "ignore_duplicates", False) |
There was a problem hiding this comment.
def check_duplicate_checksums(self, data_df, hash_col, ignore_duplicates=False): """ Detect duplicate checksum values and optionally block execution.
It is defined under the RollCall module as function check_duplicate_checksums
| # Validate and handle existing output directory | ||
| img_dir = args.output_dir | ||
| source_df, filtered_df = check_existing_images(csv_path, img_dir, source_df, filename_col, subfolders) | ||
| source_df, filtered_df = roll_call.check_existing_images(csv_path, img_dir, source_df, filename_col, subfolders) |
There was a problem hiding this comment.
Is the idea of checking for duplicate hashes in the source DataFrame first to alert the user to their existence even if they have most images already?
There was a problem hiding this comment.
Yes, the idea is to check duplicate hash values in the source DataFrame before the download step
| # If duplicates exist, preview or save | ||
| self._preview_or_save("duplicate_checksums", dupes) | ||
|
|
||
| if ignore_duplicates: |
There was a problem hiding this comment.
If this is going to run regardless of the ignore, it seems like it would make sense to print or save duplicates so the user is aware and can do something with that information.
There was a problem hiding this comment.
The current implementation prints if duplicates less than or equal to 5 else saves on the disk. I have included arg pass to ignore the duplicates checksums on the disk.
There was a problem hiding this comment.
Let's log duplicates of any number, rather than a magic threshold of 5
There was a problem hiding this comment.
Removed the 5 value threshold.
Co-authored-by: Elizabeth Campolongo <38985481+egrace479@users.noreply.github.com>
Move following functions from main.py to Validation.py + Change AGENTS.md:
validate_csv_extension
validate_output_directoryThis function was removed, as it should have been in #43.validate_filename_uniqueness
handle_missing_filenames
setup_expected_columns
RollCall Module
Added above functionalities and check_existing_images to the rollcall module which prechecks all the conditions before downloading.
Non Interactive Missing File Handling
Duplicate Checksums
Summary